1. Can we please clarify what are the use-cases we try to solve?
2. Do we want to support 64bit trace-id as a valid implementation, or this is just a transition?
3. If this is a transition problem then having a deterministic way to convert to/from may help with having independent traces.

My 2 cents:

1. If the trace-id is generated by a 64bit system then we should fill with 0 when propagating;
2. If the trace-id is received by a 64bit system instead of dropping the last 64bits which will lead to have different traces, the 64bit system may better generate a new 64bit trace-id and use some kind of linkage if they cannot pass 128bits trace-id. I would actually strongly encourage to say that for a system to be w3c compliant it requires to carry 128 bit trace-id.

@bogdandrutu You know that there exist mixed instrumentation. you have a dog in the race of instrumentation. Most people on this spec are too biased because literally same people here are developing new instrumentation they desire to be used: open-telemetry

Step away from this bias, and realize even if you wanted the world to change to your tools, you need to provide a path to that.

There are instrumentation today and for at least 3 years that are in different places, some may never change, but some may. To allow them to upgrade is better than "big bang" where you pretend entire environments can change immediately. The downgrade works with backfilling zeros.

For example, proxy can rewrite old b3 into w3c trace context mechanically by backfill zeros same as we used to upgrade people into google stackdriver format back in 2016. It is really dangerous to have zero desire to work with existing systems when designing a w3c spec. Please be reasonable and don't break what already works.

I would like to voice support for 0 backfill as opposed to random backfill as I believe it is more intuitive to understand what is going on. It is also far more compatible with 64 bit systems. I don’t really understand the downside.

Random backfill was added to make sure that systems don't take dependency on zeroes and make the best effort to propagate an entire thing even if they only use the half of the trace-id for the data recording. Specs doesn't forbid the zeroes backfill, but asks to propagate an entire thing.

I think what's misunderstood here is what the "system" is.

Random backfill is intentionally breaking, which would be surprising to have as a goal in a spec whose purpose is interop.

the propagation system is a collection of instrumentation which have different capacities. 64bit is too prevalent to either ignore or to intentionally break.

The indexing, re-fitting etc capacity of the data pipeline is out of scope for this spec, as that is about data not defined here. It is relevant to suggest that there are systems who are able to conditionally handle things in a mixed mode by looking at the 64bit part. You should know that this doesn't mean they ignore the high bits. What literally happens is that traces are grouped by 64 bits, then re-qualified by the upper bits. When the upper bits are zero (unknown) the traces are kept in a generic grouping. The "backfill random" aspect of this spec wrecks this qualification and correction heuristic. Why this was added, if to intentionally break systems today, seems literally reckless.

I understand the problem of changing backend. Spec suggesting that you can only look at subset of bytes of a trace-id. But must propagate the rest bytes thru the process. No need to save them. It is written in an assumption that tracestate must also be propagated so there is a mechanism to carry a significant number of bytes from incoming to outgoing calls. Adding a few bytes of trace-id shouldn't be a problem than.

If system makes an assumption that it will never be called by anything that will have full trace-id set with random bytes and will always be called from the zero backfill systems, than zero backfill can be used, it is not forbidden by a spec.

Let me know if I'm missing something.

what led someone to tell me about this issue was bogdan's comment here:

If the trace-id is received by a 64bit system instead of dropping the last 64bits which will lead to have different traces, the 64bit system may better generate a new 64bit trace-id and use some kind of linkage if they cannot pass 128bits trace-id. I would actually strongly encourage to say that for a system to be w3c compliant it requires to carry 128 bit trace-id.

the idea to say that nothing can use tracecontext unless every individual part of that system is 128-bit is a big problem. There's a continued misunderstanding that "propagation system" is homogeneous and it is certainly not nor likely to be for many sites. Implying you have to change all instrumentation to use tracecontext is severe and more to helping force people into open telemetry vs solving the goal of propagation header.

The other thing, and more to the point here, is that the text is very confused and misguided. Ex it says 64 bit system should add random data to make believe it is not. This hints at a control that doesn't exist. Also the example is backwards 23ce929d0e0e47360000000000000000 should be 000000000000000023ce929d0e0e4736. It is more simple to discuss what happens when you receive data, and that there are cases when you convert into or out of 64 bit trace ID carriers. No need to dance around it, just mention it, and please do not suggest anything strange. just backfill left zeroes when converting, and know that this may happen when parsing.

As a user, I think backfilling the traceID with a random trace is counter intuitive and confusing for people debugging issues in production.

More so, backfilling from the rightside insted of the left side is not only counter intuitive but unnatural, given that we are talking about numbers (in a hex representation) and when a number need to fulfill a certain length we add 0 on the left because it does not impact the actual value and I believe (again as a user) that standards should not be counterintuitive nor divorced from reality.

I think the requirement to “always propagate" leaves a lot of legacy systems in the dust, even though they could've been partially compatible with left 0 padding. I could even wrap a legacy app that uses 64bit zipkin or Jaeger header in a sidecar and automatically rewrite those headers into w3c traceparent with empty tracestate.

Left 0 padding gives a clear signal that the transaction passed through a legacy layer. Randomization requirement makes it impossible. I think randomization requirement does more harm than good (the good part isn't even that clear to me).

I would even add that if a modern 128bit capable system generates a new trace id, it MUST ensure that the first 64bits are not zeros.

We balancing many requirements and strive to encourage interoperability. I've added two more sections into rationale doc and expanded the paragraph. Please comment - it must be easier to leave comment to the document than in PR description.

This paragraph came from the discussion with NewRelic. @victorNewRelic can you please comment on your experience of a transition

I added review comments on the diffs as requested. I agree with the arguments of others in this PR that backfilling with zeros is both more compatible by default with existing brown field systems and more intuitive.

I've seen other arguments here that essentially say backfilling with randomness isn't a big deal, because it's only a suggestion and not a spec requirement. That's not cover for unnecessary interoperability breakage by default. One of my review comments talks directly to this:

The reality is that implementors will see SHOULD, and the language around 64 bit trace IDs being "violations", and they will blindly backfill with random 64 bits. As a result we'll see breakage by default instead of interoperability by default.

If this spec is truly about maximizing interoperability, I don't see how we could pick random-backfilling instead of zeros-backfilling.

Sorry for the late addition. We are actively implementing this, and I would prefer zeros as well.

The reason has to do with split traces. If Service A calls two services, B and C, with a 64-bit trace, and both B and C each backfill with different random bits, then the trace is now split, as I now have two separate trace IDs. SO the trace has become split.

Now, consider that the analysis system consuming this data accepts 128-bit IDs. When the analysis system receives 64-bit IDs directly from A, it must likewise pad the IDs with zeros. I do not see another choice, when some services only report 64 bits, and others report 128.

This is a serious issue for us. However, if there is a clear definition in the spec for how to backfill with zeros, this solves these split-trace problems very cleanly. Unfortunately, I see no solution where every service pads with random bits; currently I am watching it create a host of issues as I try to implement it.

The reason has to do with split traces. If Service A calls two services, B and C, with a 64-bit trace, and both B and C each backfill with different random bits, then the trace is now split, as I now have two separate trace IDs. SO the trace has become split.

If B and C could both update tracestate by clarifying that the original 64-bit trace has been backfilled (so the backend can use this additional hint to avoid interpreting this as a split), would this solve the problem?

@reyang unfortunately no, as information placed in tracestate is not reported by any system we interoperate with. Also, there are scenarios where we must switch headers - to propagate to a service configured with B3, for example - and thus there is no tracestate.

@nicmunroe hi! Glad to see you. Thank you for review!

@tedsuo I think you are missing a point. First, behavior when system recieves a trace is not specified. Random backfill is only for new trace-id. Second, it is recommended that B and C will propagate extra bytes. If you talking about any of opentracing implementations - there is probably some sort of baggage mechanism, so propagating of extra 64 bits shouldn't be a major problem.

Please let me know what you think

@adriancole

new relic's tracing offering is one data point, but as far as I know it is
a relatively new product and closed source. I still don't understand why
years of existing OSS practice should be ignored over this.

Absolutely! And I'm glad to see this discussion here. I only want them to share their experience of migrating to 128 bits as this entire paragraph was added after discussion of NewRelic's implementation of trace context.

Summary

1. Most people here would prefer zero backfilling instead of random backfilling.
2. The main reason why it is in the spec is that it makes it more explicit that the id should be propagated as 128bits avoiding a shortcut where systems will just start sticking to 64 and ignore the rest.
3. There might be more use cases that would justify random backfill but they don't seem to outweigh the reasons to backfill with zeros.

Is this correct so far?
I think this would be a normative change @SergeyKanzhelev?

First, behavior when system recieves a trace is not specified. Random backfill is only for new trace-id.

@SergeyKanzhelev After responding to a few other comments, this really caught my eye, and may be one of the main reasons people in this PR are having trouble hearing each other. I'm not sure you can specify one thing (how to generate the trace ID) without considering the other (how to receive a 128 bit trace when you're a 64 bit system, and then how to propagate it again later).

Why? If a 64 bit system receives a 128 bit trace ID and downgrades it to 64 bit (assuming that's the best it can do), then what is it supposed to do when it goes to propagate it on outbound calls? The spec is very clear about all trace IDs being 128 bit, and this section talks about backfilling with random values. So they'll probably backfill with random values when they propagate downstream. This is really bad, as @tedsuo 's example showed.

It doesn't matter that there's a small side note buried in the text saying "it's only SHOULD, you can backfill with zeros if you want" - implementors who are trying to be good citizens will backfill with random, because it's what the spec recommends. And it doesn't matter that the random backfill stuff is only supposed to be for ID generation - the 64 bit system has to do something to conform to the w3c 128 bit trace ID requirement when propagating, so it's logical that they would follow the section talking about what 64 bit systems are supposed to do with trace IDs.

If, on the other hand, the recommendation was to backfill with zeros both for generation and 64-bit downgrade+propagation, then you'd get some level of interoperability and compatibility by default. You'd get this base level of interoperability even if a trace bounced back and forth through multiple 64 bit and multiple 128 bit systems on a single request.

@nicmunroe thank you for the note. I'm also trying to understand how to make sure we are not talking pass each other.

Yes, spec is only talking about new ID generation. In case of propagation - either fill up with zeroes or try to pass by extra bytes unchanged if you can. It is definitely not clear from the text as it raises so much confusion. This is how this PR is actually started.

As for the inter-oparability and left padding - the prior art reference goes to how, for example, getTrace works on Zipkin:

When strict trace ID is disabled, spans with the same right-most 16 characters are returned even if the characters to the left are not.

So asking for left padding of randomness breaks this scenario. You'd want some randomness in the right part of a trace so inter-operability with 64bit systems would work.

@danielkhan as I see it - in an order of importance there are three questions:

1. Backfilling on left vs. right
2. Left padding of randomness.
3. Zero vs. random backfill

1st is not specified in a spec. So we can add it without normative change to the spec.

We have a strange SHOULD-MUST combination of randomness generation. Reading it is unclear what it actually asks for. One option is to nuke it completely from the spec. Again, I'd argues not a normative change as the entire paragraph is confusing.

For the item 3 "Zero vs. random backfill" on a new trace-id generation - I'd suggest to keep randomness. With the first two items addressed inter-operability in terms of downgraded experience will be addressed. And randomness will improve inter-operability in terms of upgrading experience.

If we all agree with this - there will be (from my perspective) no normative changes and everybody's interests will be addressed.

I recommend filing separate issues and holding a discussion / vote there before proposing a PR to change.

For the item 3 "Zero vs. random backfill" on a new trace-id generation - I'd suggest to keep randomness. With the first two items addressed inter-operability in terms of downgraded experience will be addressed.

I think most people on this thread disagree with this. If a request goes through several 64bit systems, then zero-padding on egress from those systems allows almost the whole request to be associated with a single trace ID. Random padding will break it into many traces, with no way of correlating.

I agree with @yurishkuro. I hardly felt heard in this discussion, and I do not see my concerns being clearly addressed. I would like to see this hashed out more fully.

@tedsuo so you mentioning two issues. First, split trace. In this example you assume that B and C will fill up trace-id with zeroes:

The reason has to do with split traces. If Service A calls two services, B and C, with a 64-bit trace, and both B and C each backfill with different random bits, then the trace is now split, as I now have two separate trace IDs. SO the trace has become split.

When spec is saying:

1. "on new trace-id generation"

When a system operates with a trace-id that is shorter than 16 bytes, on new trace-id generation it SHOULD fill-in the extra bytes with random values rather than zeroes.

2. On propagation:

In situations, when it is absolutely impossible to propagate the entire trace-id to the downstream components, but only a subset of the original trace-id will be propagated, system SHOULD fill up extra bytes with zeroes.

Second concern is about analytical system. And rationale doc addresses this concern saying that:

Specification uses "SHOULD language" when asking to fill up extra bytes with
random numbers instead of zeros. Typically, tracing systems that will not
propagate extra bytes of incoming trace-id will not follow this ask and will
fill up extra bytes with zeros.

So this is not really a concern - if system breaks this spec requirement anyway, you can break one more "SHOULD". In fact spec is not addressing these type of services at all - it uses MUST langugage that the entire trace-id must be propagated even if you only recording part of it.

Furthermore if you look at the same example and assume that A that started trace is 128bit system, you still need to build an analytics system that operated on lower bytes only. So random generation on new trace-id will help.

What am I missing?

@yurishkuro

I recommend filing separate issues and holding a discussion / vote there before proposing a PR to change

Absolutely! This PR simply removes the uncertancy in the current spec w.r.t. how it was written. Mostly around the fact that random backfill is only requested for new trace-id, not the case when you propagate somebody else's trace-id.

So the way I'd suggest we approach it is by merging this PR and filing these three issues:

1. Backfilling on left vs. right
2. Left padding of randomness.
3. Zero vs. random backfill

My comment was written in response to @danielkhan's summary trying to break down the problem in smaller issues.

@SergeyKanzhelev "random backfill is only requested for new trace-id" kind-of doesn't make sense to me as a use case. If a system is truly 64bit, and not upgradeable, it most likely won't have any place to store the other random part of the trace. The backfill only applies on egress from a 64bit system, which is where many are arguing a zero-padding is better than random padding. Note that the padding may be done not by the 64bit system itself, but by a sidecar that rewrites the headers.

@yurishkuro these systems

it most likely won't have any place to store the other random part of the trace.

breaks a lot (like not propagating tracestate) and can back fill with zeros on trace-id generation.

Spec uses MUST language on propagation with a small not on what to do when it's absolutely impossible.

However, once the systems is capable to propagate all 128 bit (even though 64 of these bits may not be recorded), the ask is to do random generation. For downstream 64bit only systems these "slightly upgraded" 64 bit systems may look like a truly 128 systems that still needs to be supported.

Are you suggesting to add another issue for discussion:

4. Spec must address 64 bit systems that doesn't have propagation mechanism and remove MUST language on the whole trace-id propagation for these systems.

Also I realized we may need a names to differentiate systems:

• 128bit system - system operating with 128bit trace-id (fully compliant).
• 64bit system - system operating with 64bit trace-id. Not capable to propagate tracestate and 64 extra bits of trace-id
• 128on64bit system - system that records 64 bit trace-id. But capable to propagate extra 64 bit of trace-id from incoming request to outgoing request.

so spec mostly talking about 128on64 systems. And have a small note about 64bit system:

when it is absolutely impossible to propagate the
entire trace-id to the downstream components, but only a subset of the
original trace-id will be propagated, system SHOULD fill up extra bytes with
zeroes.

Spec must address 64 bit systems that don't have propagation mechanism and remove MUST language on the whole trace-id propagation for these systems.

I think so. Most of the feedback on this thread was about "legacy" systems. If a system can be upgraded to support tracestate propagation, it might as well be upgraded to support full 128bit trace-id, I don't see much difference. The concern is that upgrading is not feasible in many cases.

I think so. Most of the feedback on this thread was about "legacy" systems. If a system can be upgraded to support tracestate propagation, it might as well be upgraded to support full 128bit trace-id, I don't see much difference. The concern is that upgrading is not feasible in many cases.

Good. We are on the same page here.

Spec must address 64 bit systems that doesn't have propagation mechanism and remove MUST language on the whole trace-id propagation for these systems.

@SergeyKanzhelev to be clear, this is the scenario I am raising. I believe the confusion relates to whether existing systems had somewhere to put the extra 64 bits. The 64bit, 126bit, 64on128 distinction helps. Thank you @yurishkuro for making the case for handling legacy 64bit systems.

Furthermore if you look at the same example and assume that A that started trace is 128bit system, you still need to build an analytics system that operated on lower bytes only. So random generation on new trace-id will help.

I admit I don't see the logic here. As long as we are accepting data from 64bit services, our analytics system will have to operate with that amount of randomness in the ID. How would generating new random IDs on the client side be helpful? Not saying that it is unhelpful, just that it is not clear to me.

So the way I'd suggest we approach it is by merging this PR and filing these three issues:

Backfilling on left vs. right
Left padding of randomness.
Zero vs. random backfill

Thank you. Having looked at it from several directions now, I don't believe there is a complete solution for backwards compatibility, except to stick with only 64 bits in the trace id until every 64bit service in the trace is upgraded to, at minimum, 128on64bit mode. Since this kind of compatibility is turning out to be critical for a number of users, I believe it would be helpful for the spec to provide guidance on this issue.

I'm closing this PR in favor of this issue: #349 Thank you everybody for a valuable feedback! Let's make sure the spec is fixed to address compatibility scenarios identified in this discussion.